Implement table & tree disk usage statistics#17169
Implement table & tree disk usage statistics#17169
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements comprehensive disk usage statistics collection for both Tree Model (device-based) and Table Model databases in IoTDB. The implementation provides monitoring capabilities at the table/device level and time partition level.
Changes:
- Adds
SHOW DISK_USAGESQL statement for Tree Model with on-demand calculation - Implements
table_disk_usageinformation schema table for Table Model with persistent cache - Introduces background task infrastructure for cache maintenance with periodic compaction
- Adds predicate push-down and limit/offset optimization support for information schema tables
Reviewed changes
Copilot reviewed 95 out of 95 changed files in this pull request and generated 33 comments.
Show a summary per file
| File | Description |
|---|---|
| pom.xml | Updates tsfile version to 2.2.1-260205-SNAPSHOT |
| TsFileID.java | Adds SHALLOW_SIZE constant (contains bug) |
| InformationSchema.java | Adds table_disk_usage schema and push-down support (contains bug) |
| TableDiskUsageCache*.java | Core cache implementation with writer/reader classes |
| ShowDiskUsageNode.java | Plan node for tree model disk usage queries |
| TableDiskUsageInformationSchemaTableScanNode.java | Plan node for table model information schema scans |
| ShowDiskUsageOperator.java | Execution operator for tree model |
| DiskUsageStatisticUtil.java | Base utility class for disk usage calculation |
| IoTDBDescriptor.java | Configuration support (contains bug) |
| Integration tests | Comprehensive tests for both tree and table models |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java
Outdated
Show resolved
Hide resolved
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
Outdated
Show resolved
Hide resolved
...b/storageengine/dataregion/utils/tableDiskUsageCache/TimePartitionTableSizeQueryContext.java
Outdated
Show resolved
Hide resolved
...ore/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/TsFileID.java
Outdated
Show resolved
Hide resolved
...core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java
Show resolved
Hide resolved
...ode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TableDiskUsageTest.java
Show resolved
Hide resolved
...ode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TableDiskUsageTest.java
Show resolved
Hide resolved
.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java
Show resolved
Hide resolved
.../java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java
Show resolved
Hide resolved
...b/storageengine/dataregion/utils/tableDiskUsageCache/object/IObjectTableSizeCacheReader.java
Show resolved
Hide resolved
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #17169 +/- ##
============================================
+ Coverage 39.47% 39.57% +0.09%
Complexity 282 282
============================================
Files 5098 5120 +22
Lines 341273 343782 +2509
Branches 43471 43795 +324
============================================
+ Hits 134714 136041 +1327
- Misses 206559 207741 +1182 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 99 out of 99 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| keyFile1.createNewFile(); | ||
| valueFile1.createNewFile(); | ||
| tempKeyFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
| keyFile1.createNewFile(); | ||
| valueFile1.createNewFile(); | ||
| tempKeyFile2.createNewFile(); | ||
| tempValueFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
|
|
||
| keyFile1.createNewFile(); | ||
| valueFile1.createNewFile(); | ||
| keyFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
| keyFile1.createNewFile(); | ||
| valueFile1.createNewFile(); | ||
| keyFile2.createNewFile(); | ||
| valueFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
|
|
||
| keyFile1.createNewFile(); | ||
| valueFile1.createNewFile(); | ||
| keyFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
| keyFile1.createNewFile(); | ||
| valueFile1.createNewFile(); | ||
| keyFile2.createNewFile(); | ||
| tempValueFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
| + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX); | ||
|
|
||
| keyFile1.createNewFile(); | ||
| tempKeyFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
|
|
||
| keyFile1.createNewFile(); | ||
| tempKeyFile2.createNewFile(); | ||
| tempValueFile2.createNewFile(); |
There was a problem hiding this comment.
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
| context.getTimePartitionTableSizeQueryContextMap().entrySet()) { | ||
| Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap(); | ||
| for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) { | ||
| int i = Integer.parseInt(tableSizeEntry.getKey().substring(5)); |
There was a problem hiding this comment.
Potential uncaught 'java.lang.NumberFormatException'.
|




Description
This PR implements disk-usage statistics collection in table level and device level for tree model. It adds the necessary data structures, background tasks, and read APIs to compute and expose disk usage metrics used by monitoring, admission control and operational tooling.
Tree Model (No Cache)
Implements ShowDiskUsageNode and ShowDiskUsageOperator.
• Disk usage is calculated by scanning relevant TsFiles at query time.
• Supports:
• Path pattern matching
• Time partition filtering
• Existing SQL semantics (SHOW DISK_USAGE)
Table Model (With Cache)
Table Model introduces a dedicated disk usage cache:
• TableDiskUsageCache manages all cache operations.
• A single-threaded background worker processes write, read, and maintenance tasks via an operation queue.
• Cache state is persistent across restarts.
Cached Data
• TsFile-level table size statistics
• Object file size deltas, recorded incrementally
• Periodic snapshot + delta compaction
Query Integration
• Exposes statistics via information_schema.table_disk_usage.
• Supports:
• Predicate pushdown (except on aggregated size columns)
• Limit / offset
• Parallel region-level scanning